Intelligent network operation platform for network fault mitigation

ABSTRACT

Embodiments of the present disclosure provide systems, methods, and computer-readable storage media that leverage artificial intelligence and machine learning to identify, diagnose, and mitigate occurrences of network faults or incidents within a network. Historical network incidents may be used to generate a model that may be used to evaluate real-time occurring network incidents, such as to identify a cause of the network incident. Clustering algorithms may be used to identify portions of the model that share similarities with a network incident and then actions taken to resolve similar network incidents in the past may be identified and proposed as candidate actions that may be executed to resolve the cause of the network incident. Execution of the candidate actions may be performed under control of a user or automatically based on execution criteria and the configuration of the fault mitigation system.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Indian Provisional ApplicationNo. 202041026007 filed Jun. 19, 2020, and entitled “INTELLIGENT NETWORKOPERATION PLATFORM FOR 5G NETWORKS,” the content of which isincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present invention relates generally to network fault resolutiontechniques and more specifically to systems and methods that leveragemachine learning and artificial intelligence to rapidly identify,diagnose, and mitigate causes of network faults.

BACKGROUND OF THE INVENTION

Network technology advancements have resulted in rapid deployment andadoption of network services and functionality. For example, theservices and functionality provided by voice communication networks(e.g., 4th Generation (4G) and 5G communication networks), contentdistribution networks, enterprise and cloud-services networks, and thelike have become widespread and play a vital role in the way we work,communicate, and consume audio and video content. However, the expandedcapabilities of such networks due to these advancements are not withoutdrawbacks. For example, networks have become increasingly complex, oftenincorporating many different types of devices, topologies, communicationprotocols, and hardware from different vendors, which has made it moredifficult to diagnose and resolve any network incidents (e.g., faults,errors, loss of services, and the like). When these network incidentsoccur, services provided by the network(s) may be unavailable orfunction improperly, creating problems for the users and devicessupported by the network(s).

Many complex networks today are managed from a network operations center(NOC), which may be a centralized location from which networkadministrators manage, control and monitor one or more networks and theservices those networks provide. While existing technologies providemechanisms to detect network incidents when they occur, handling thelarge number of alarms raised by network incidents in today's complexnetworks presents significant challenges with respect to determining thecauses of each network incident and how each incident should beresolved. Presently available techniques often rely on excessivemanpower (e.g., allocating many individuals to diagnose and resolvenetwork incidents) and static knowledge databases to determine how toresolve network incidents and ultimately implement the determinedsolution(s) for each individual alarm. Often the overall processconsumes long durations of time in order to resolve each networkincident.

The problems described above present significant challenges with manynewer network technologies, such as 5G networks. As a result, improvedtechniques for identifying and resolving network incidents are needed,especially considering that these newer network technologies areincreasingly being used to support mission critical applications thatdemand high availability of the networks. For example, as 5G networkdeployments expand, it is expected to bring about a 50-60% increase intotal number of network related incidents (e.g., due, at least in partto increased deployment of devices relying on machine-to-machine (M2M)communications and Internet of Things (IoT) devices). Legacy solutionsfor resolving such network incidents may lead to severe servicedegradations and take significant amounts of time to resolve.Additionally, previous networks (e.g., 3G, 4G/LTE, networks, etc.) mayexperience less noise than next generation networks. One reason morenoise may occur in next generation networks is that those networks mayrely more on virtualization of network functionality and services ascompared to previous networks. This increased noise may make it moredifficult to diagnose and take corrective actions when network incidentsoccur.

BRIEF SUMMARY OF THE INVENTION

The present invention is directed to systems, methods, andcomputer-readable storage media that leverage artificial intelligenceand machine learning techniques to create and train models that may beused to evaluate network incidents and determine actions that may beperformed to resolve the network incidents. Historical network incidentdata may be analyzed using artificial intelligence processes to identifyclusters of network incidents having similarities. The clusters may beused to generate models of the historic network incident data that maybe used to identify root causes of network incidents. Based on theidentified cause(s) of the network incident, historic network incidentresolutions may be evaluated to identify candidate actions that may betaken to resolve network incidents in real-time.

Machine learning processes may be used to evaluate the candidateactions, such as to assign a score to the candidate action(s) and/or toclassify the candidate action(s) into one of a plurality of categories.The score and/or the classification may be used to resolve the networkincident. For example, when a score associated with an action satisfiesa threshold score, embodiments may automatically execute the actionpredicted to resolve the network incident. Where the score does notsatisfy the threshold score (or in embodiments where automatic executionis not utilized or utilized in a more limited fashion), a notificationidentifying the one or more candidate actions may be transmitted to auser (e.g., an information technology (IT) or network administrator) andpresented in a graphical user interface. The graphical user interfacemay include interactive elements that allow the user to execute at leastone of the candidate actions, where the actions may be performed remoteto the node or portion of the network where the network incidentoccurred, such as from a NOC supporting the network.

As actions are executed, the system may monitor the network to ensurethat the action(s) resolved the network incident. If an action did notresolve the incident, additional candidate actions may be executed untilthe network incident is resolved. Information associated with networkincidents resolved using the techniques disclosed herein may be providedas feedback that may be incorporated into the historical data and usedto train the models and machine learning/artificial intelligenceprocesses so that future network incidents may be more rapidly diagnosedand resolved.

The foregoing has outlined rather broadly the features and technicaladvantages of the present invention in order that the detaileddescription of the invention that follows may be better understood.Additional features and advantages of the invention will be describedhereinafter which form the subject of the claims of the invention. Itshould be appreciated by those skilled in the art that the conceptionand specific embodiment disclosed may be readily utilized as a basis formodifying or designing other structures for carrying out the samepurposes of the present invention. It should also be realized by thoseskilled in the art that such equivalent constructions do not depart fromthe spirit and scope of the invention as set forth in the appendedclaims. The novel features which are believed to be characteristic ofthe invention, both as to its organization and method of operation,together with further objects and advantages will be better understoodfrom the following description when considered in connection with theaccompanying figures. It is to be expressly understood, however, thateach of the figures is provided for the purpose of illustration anddescription only and is not intended as a definition of the limits ofthe present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, reference isnow made to the following descriptions taken in conjunction with theaccompanying drawings, in which:

FIG. 1 shows a system for diagnosing and resolving network incidentsaccording to embodiments of the present invention;

FIG. 2 shows a screenshot of an interface displaying a graphicalrepresentation of historical incident data according to embodiments ofthe present invention;

FIG. 3 shows a screenshot illustrating a graphical representation ofclustered historical incident data according to embodiments of thepresent invention; and

FIG. 4 is a block diagram illustrating aspects of fault mitigationprocessing according to embodiments of the present invention;

FIG. 5 is a flow diagram of a method for diagnosing and resolvingnetwork incidents according to embodiments of the present invention.

It should be understood that the drawings are not necessarily to scaleand that the disclosed embodiments are sometimes illustrateddiagrammatically and in partial views. In certain instances, detailswhich are not necessary for an understanding of the disclosed methodsand apparatuses or which render other details difficult to perceive mayhave been omitted. It should be understood, of course, that thisdisclosure is not limited to the particular embodiments illustratedherein.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present disclosure provide techniques for rapididentification of network incident causes and diagnosis of actions thatmay be executed (e.g., locally or remotely) to mitigate the causes ofthe network incidents. Using machine learning and artificialintelligence techniques, embodiments leverage historic network incidentsdata to determine the action(s) that may be executed to resolve thenetwork incident cause. The network may be monitored for a period oftime after execution of the action to ensure the cause of the networkincident is resolved and additional actions may be executed if previousactions were unsuccessful in mitigating the cause of the networkincident. The concepts disclosed herein may enable network faults to beresolved more rapidly and allow emerging network incidents (e.g.,network incidents that have not previously occurred or occurinfrequently) to be identified, resolved, and taken into considerationin the future, which may play a critical role in resolving issues thatoccur as network technologies continue to evolve and become morecomplex.

Referring to FIG. 1, a system for diagnosing and resolving networkincidents in accordance with aspects of the present disclosure is shownas a system 100. As shown in FIG. 1, the system 100 includes a faultmitigation device 110. The fault mitigation device 110 may becommunicatively coupled to one or more networks 130. Additionally, thefault mitigation device 110 may be communicatively coupled to one ormore user devices 140 via the one or more networks 130. The faultmitigation device 110 may be configured to monitor at least one networkof the one or more networks 130 for faults (e.g., network errors orissues that may result in degraded network performance, loss of networkservices, or other issues) and initiate operations to mitigate anydetected faults. Operations to mitigate detected faults may includeproviding recommendations associated with one or more actions configuredto resolve or correct a detected fault to the user device(s) 140, whichmay be devices operated by information technology (IT) or networkadministration personnel responsible for maintaining the network and thenodes supporting the network. In aspects, operations to mitigate thedetected faults may also include automatic correction of detected faultsdetected by the fault mitigation device 110. Additional detailsregarding the above-mentioned features and functionality of the faultmitigation device 110 are described in more detail below.

The fault mitigation device 110 includes one or more processors 112,clustering logic 114, modelling logic 116, one or more input/output(I/O) devices 118, and a memory 120. Each of the one or more processors112 may be a central processing unit (CPU) having one or more processingcores or other computing circuitry (e.g., a microcontroller, one or moreapplication specific integrated circuits (ASICs), and the like). Thememory 120 may include read only memory (ROM) devices, random accessmemory (RAM) devices, one or more hard disk drives (HDDs), flash memorydevices, solid state drives (SSDs), network attached storage (NAS)devices, other devices configured to store data in a persistent ornon-persistent state, or a combination of different memory devices. Thememory 120 may store instructions 122 that, when executed by the one ormore processors 112, cause the one or more processors 112 to perform theoperations described in connection with the fault mitigation device 110with reference to FIGS. 1-4. Additionally, the memory 120 may store oneor more databases 124 that support operations of the fault mitigationdevice 110. Exemplary aspects of the types of information that may bestored in the one or more databases and how that information may be usedby the fault mitigation device 110 are described in more detail below.

The I/O devices 118 may include one or more display devices, a keyboard,a stylus, one or more touchscreens, a mouse, a trackpad, a camera, oneor more speakers, haptic feedback devices, or other types of devicesthat enable a user to receive information from or provide information tothe fault mitigation device 110. Additionally, the I/O devices 118 mayinclude one or more communication interfaces configured tocommunicatively couple the fault mitigation device 110 to the one ormore networks 130 via wired or wireless communication links according toone or more communication protocols or standards (e.g., an Ethernetprotocol, a transmission control protocol/internet protocol (TCP/IP), aninstitute of electrical and electronics engineers (IEEE) 802.11protocol, and an IEEE 802.16 protocol, a 3rd Generation (3G)communication standard, a 4th Generation (4G)/long term evolution (LTE)communication standard, a 5th Generation (5G) communication standard,and the like).

In an aspect, the fault mitigation device 110 may be provided as part ofa network operations center (NOC) infrastructure providing a centralizedframework for managing networks and network nodes, which may includepublic and private networks operated by an entity (or group of entities)and that may span across different geographic regions. It is noted thatwhile described with reference to FIG. 1 as a “device,” thefunctionality provided by the fault mitigation device 110 may also beimplemented via software (e.g., the instructions 122) running on the NOCinfrastructure or as a software layer that sits between the NOCinfrastructure and the network.

In aspects, the one or more networks 130 may include a plurality ofnodes, such as nodes 132, 134, 136, 138. The nodes of the network mayinclude physical devices, such as routers, relays, switches, basestations (e.g., evolved node-Bs (eNBs), next generation node-Bs (gNBs),femtocells, picocells, etc.), servers (e.g., data servers, web servers,etc.), security appliances, user devices, or other devices configured toprovide functionality and services to one or more end users (e.g.,employees of an organization, consumers accessing websites,communication service subscribers, and the like). The nodes of the oneor more networks 130 may also include virtual nodes (e.g., virtualnetwork devices providing functionality and/or services to the users ofthe network(s)). It is noted that the nodes monitored by the faultmitigation device 110 may be nodes operating within a network controlledby an entity that operates the fault mitigation device 110 but that theone or more networks 130 may include other nodes that are part of othernetworks. For example, an organization may have LANs, WANs, and othertypes of network infrastructure configured to connect nodes of theorganization to each other to facilitate intra-organizationcommunication, but the organization's networks and nodes may alsotransmit and receive communications over external networks and nodes,such as networks and nodes providing the organization with access to theInternet or other public communication networks and services. It isnoted that the fault mitigation and mitigation techniques describedherein may be capable of detecting whether faults are the result ofnodes within an organization's network, which the fault mitigationdevice 110 may provide functionality and processes to correct, andfaults that occur due to failures and issues arising in nodes externalto the organization's network nodes.

The one or more user devices 140 may include computing devicesassociated with an entity's network administration or IT personnel. Theuser devices 140 may include desktop computing devices, laptop computingdevices, smartphones, personal digital assistants (PDAs), tabletcomputing devices, or other types of devices operable to perform theoperations described herein with reference to the one or more userdevices 140. It is noted that such computing devices may include one ormore processors, memory, I/O devices, or other components supporting thefunctionality and operations described herein.

In aspects, the system 100 may also include other devices 150. The otherdevices 150 may include Internet of things (IoT) devices, vehicles(e.g., cars, trucks, boats, planes, etc.), or other types of devicesthat may include a communication capability. In an aspect, one or moreof these types of devices may be combined. To illustrate, a vehicle mayinclude IoT devices, such as sensors, that generate data related tooperations of the vehicle (e.g., speed, temperature, tire pressure,location data, or other types of information) and the generated data maybe transmitted to an external system (e.g., the fault mitigation device110) via the one or more networks 130. The fault mitigation device 110may be configured to receive data from the other devices 150 and utilizethat data to identify, diagnose, and correct faults within a network, asdescribed in more detail below.

As briefly described above, the fault mitigation device 110 may beconfigured to detect and mitigate (e.g., diagnose and resolve) faultsoccurring within at least one network of the one or more networks 130.When a fault occurs in a network, an alarm message may be generated(e.g., by one or more network monitoring devices) and transmitted tofault mitigation device 110 or may be transmitted to the NOCinfrastructure and intercepted or detected by the fault mitigationdevice 110. The alarm message may include one or more parametersassociated with the fault that triggered generation of the alarm and maybe generated by one or more of the nodes 132, 136, 134, 138 (or othernodes of the one or more networks 130), by the user device 140, or bythe other device(s) 150. It is noted that in some instances multiplealarm messages may be generated for a single fault. To illustrate, if abase station of a cellular network (e.g., an evolved nodeB (eNB) of a4G/LTE network or a next generation nodeB (gNB) of a 5G network) goesdown, neighboring base stations (e.g., nodes of the network) maygenerate alarm messages indicating one of their neighboring basestations is unavailable. Additionally, the user device(s) 140 and otherdevices 150 served by the base station that went down may also generatealarm messages that may be received by the fault mitigation device 110.The base station may also generate an alarm message. Each of these alarmmessages may be received by the fault mitigation device 110 and used todetect, diagnose, and mitigate the fault within the network (e.g., thecause of the base station going down). It is noted that some of thealarm messages may be related to symptoms of the fault, such as thealarm messages generated by the devices served by the base station, andsome of the alarm messages may be related to the cause of the fault,such as the alarm message received from the base station.

In an aspect, the parameters of the alarm message(s) may include a nodeparameter, an agent parameter, a summary parameter, an alert keyparameter, and an alert group parameter. The node parameter may identifyan entity or node from which the alarm originated, such as to identifyone of the nodes 132, 134, 136, 138. The agent parameter may includeinformation associated with a sub-manager that generated the alarm. Forexample, the nodes 132, 134, 136, 138 may include functionalityconfigured to monitor the state of different aspects of the nodes. Themonitoring functionality may include monitoring backhaul networkconnectivity of the node (e.g., a status of connectivity to the backhaulnetwork), neighbor node monitoring (e.g., monitoring connectivity of oneor more neighbor nodes), signal quality monitoring (e.g., monitoring aquality of signals transmitted by the node, which may be based onchannel estimates received from served devices or other techniques),node performance monitoring (e.g., monitoring performance parameters ofa node, such parameters associated with performance of CPUs, memory,etc.), node interface monitoring (e.g., monitoring individualports/interfaces of a node, such as a X2 interface, a S1 interface,etc.), or other types of monitoring functionality. The monitoringfunctionality may be provided via software agents running on one or moreprocessors of the nodes and/or dedicated hardware based monitoringdevices. When a problem is detected with the operations of a node, themonitoring functionality may generate the alarm message, which mayidentify the agent/hardware device generating the alarm message (ortriggering generation of the alarm message) in the agent parameter.

The summary parameter may include information representative of thealarm condition and one or more managed object instances affected by thealarm condition. For example, where a base station experiences ahardware failure, the summary parameter may include information thatindicates a hardware component (e.g., a transmitter, etc.) has failed,and where the base station experiences a service or software failure,the summary parameter may include information that indicates the serviceor software process that failed. In an aspect, each alarm type may bedefined in a Management Information Base (MIB) of a device typebelonging to a particular vendor and may be configured in the nodes ofthe network. Based on a specific Object Identifier (OID) generated fromthe node, the monitoring agent populates the alarm summary field withrelevant values. The alert key parameter may indicate the managed objectinstance referenced by the alarm message, and the alertgroup may includeinformation descriptive of the failure type indicated by the alarmmessage. In an aspect, the alert key may contain a descriptive key thatindicates the object instance referenced by the alarm. The alert key maybe a SNMP instance of the managed object represented by the alarm.Usually, this can be obtained by extracting the instance from the OID ofone of the trap's variable bindings. The value of the alert key may beused to ensure proper deduplication of alarms. For example, a link downalarm may have an alert key defined as LINK DOWN. The alert group mayindicate the value under which a trap can be grouped, for example,AUTHENTICATION STATUS. It is noted that the exemplary parametersdescribed above have been provided for purposes of illustration, ratherthan by way of limitation and that additional parameters, differentparameters, or fewer parameters may be utilized by the conceptsdisclosed herein.

The fault mitigation device 110 may utilize the parameters of the alarmmessage to identify a cause associated with the alarm message. In anaspect, the parameters of the alarm message may be analyzed against aknown error database (KEDB) to see if the parameters match any knownerrors. If the parameters match a known error, the cause of the errormay be identified from the information stored in the KEDB and one ormore actions to resolve the cause of the error may be determined basedon the information stored in the KEDB. If the cause of the fault orerror cannot be determined based on the KEDB, artificial intelligenceprocesses may be applied to the parameters to diagnose and identify thecause of the fault and determine one or more actions to mitigate thefault.

The artificial intelligence processes may include clustering techniquesand machine learning logic configured to analyze the parameters anddetermine a cause of the fault, as well as a confidence level associatedwith the determined cause. For example, the clustering logic 114 may beconfigured to determine one or more clusters based on the parameters andhistorical data, which may be stored at a historical database of the oneor more databases 124. The historical data may include informationassociated with alarm conditions previously experienced within therelevant network over a period of time (e.g., 1 month, 6 months, 1 year,or some other time period). The clustering logic 114 may be configuredto determine the clusters based on a nearest neighbors algorithm that isconfigured based on the parameters of the alarm message. Duringclustering, network incidents represented by the historical data may bearranged into clusters by the nearest neighbors algorithm of theclustering logic 114 based on a percentage of similarity in incidentfields. Each incident in the historical data may include informationassociated with data similar to the parameters of the alarm message andthe different clusters may correspond to incidents sharing the same orsimilar parameters.

To illustrate, a first cluster may correspond to network incidents orfaults sharing similar parameters, which may be indicative of a firsttype or group of network incidents or faults, and a second cluster maycorrespond to network incidents sharing a different set of parametersimilarities, which may be indicative of a second type or group ofnetwork incidents or faults. The different clusters may then be analyzedto determine which cluster is closest to the parameters of the receivedalarm message. For example, historic network incidents of a clustersharing many similarities with respect to the parameters of a receivedalarm message may indicate that the alarm message is related to a causeof the historic network incidents associated with the cluster. Theparameters of a received alarm/fault may be segregated within individualfields and the combination of such parameters may be compared acrossparameters of alarms/faults included in historical data. The comparedalarms may be grouped together based on matching combinations ofparameters to form unique clusters. Once the alarms are clustered, aknown error database (KEDB) may be referenced to identify the cause ofthe alarm/fault on the basis of root cause analysis that was carried outfor previous occurrences of alarms/faults sharing the same or similarsets of parameters. It is noted that terms utilized to describe theclustering logic such as “nearest neighbors” and “closest” are notintended to convey geographic distance or proximity and are instead usedto describe the similarity between parameters of the network incidentsor faults represented by the model created by the clustering logic 114and/or similarities between the network incidents or faults representedby the model and the parameters of a received alarm message. Forexample, network incidents or faults represented by the differentclusters may be associated with network nodes that are located at asingle location or at many different geographic locations.

Referring briefly to FIGS. 2 and 3, screenshots illustrating exemplaryaspects of clustering network incidents are shown. In FIG. 2, ascreenshot of an interface displaying a graphical representation ofhistorical incident data is shown at 210. The graphical representationshown in FIG. 2 may be part of a graphical user interface that may bepresented to a user (e.g., a user of the user device 140 of FIG. 1) andmay provide one or more controls 120 for interacting with andmanipulating the graphical representation. In FIG. 3, a screenshotillustrating a graphical representation of clustered historical incidentdata is shown and includes two exemplary clusters 310, 320. The cluster310 may correspond to a first type of network incident while the cluster320 may correspond to a second type of network incident. To illustrate,the cluster 320 may correspond to network incidents that occurred as theresult of a configuration change to one or more network nodes while thecluster 310 may correspond to network incidents that were caused bysomething other than a configuration change. The graphicalrepresentation shown in FIG. 3 may be part of a graphical user interfacethat may be presented to a user (e.g., a user of the user device 140 ofFIG. 1) and may provide one or more controls 330 for interacting withand manipulating the graphical representation. It is noted that the oneor more controls 220 of FIG. 2 and the one or more controls 330 of FIG.3 may include the same controls, completely different sets of controls,or a combination of common and unique controls relevant to the graphicalrepresentation being displayed.

Referring back to FIG. 1, in an aspect, the clustering may be performediteratively. For example, a first iteration may achieve a looseclustering but the historical data may not be sufficiently grouped suchthat specific clusters represent groups of similar incidents. A seconditeration may result in a finer clustering granularity with well-definedgroups starting to form. The clustering logic 114 may continue toexecute the nearest neighbors algorithm (or another clustering techniqueor combination of techniques) until well defined clusters have beenachieved with respect to the historical data (e.g., until clustersrepresent network incidents having a threshold degree of similarity).Each of the clusters identified by the clustering logic 114 mayrepresent a different fault group and the network incidents within acluster may share similarities with each other (e.g., similar networkincidents, parameters, nodes, networks, network incident causes, and thelike).

Once clustering is complete (whether multiple iterations are performedor a single iteration), the modelling logic 116 may analyze the clustersand information derived from the analysis may be compiled into a model.In an aspect, the model may be a multi-dimensional array, such as a60-dimensional array, representing similarities between networkincidents within each cluster. The model may facilitate analysis ofhistorical network incidents to identify and evaluate hidden patternswithin the historical data, such as to identify instances where asimilar type of incident could have been updated with different values.In an aspect, when an alarm is resolved by an NOC operator or fieldengineer, notes associated with the alarm may be recorded that describethe resolved error or fault and over time similar types of faults mayresult in similar types of notes, which may help with the identificationof pattern relationships and meaning of similar resolution notes for thefault occurrence, which may aid in achieving effective clustering. It isnoted that while FIG. 1 shows the clustering logic 114 and modellinglogic 116 as different logical blocks of the fault mitigation device110, it is to be understood that such depiction has been provided forpurposes of illustration, rather than by way of limitation and that thefunctionality provided by the clustering logic 114 and the modellinglogic 116 may be integrated (e.g., as clustering and modelling logic).Moreover, it should be understood that the functionality provided by theclustering logic 114 and the modelling logic 116 may be stored asinstructions, such as the instructions 122, executable by the one ormore processors 112 to perform the operations described herein withrespect to the clustering logic 114 and the modelling logic 116.

Additionally, the model may be used to identify or determine correctiveactions for a current fault experienced within the system 100, such asto identify corrective actions that may be utilized to address thenetwork incident or fault corresponding to the alarm message received bythe fault mitigation device 110. For example, the fault mitigationdevice 110 may evaluate the received alarm message to perform a rootcause analysis. The root cause analysis may result in identification ofa parent or primary alarm. Once the parent or primary alarm isidentified, the parameters included in the alarm message may beevaluated against the model to determine one or more candidate actionsthat may be performed to resolve the issues related to the alarmmessage. For example, an artificial intelligence process may analyze themodel using a nearest neighbor algorithm to find historical networkincidents corresponding to the parameters of the alarm message, asdescribed above. Actions corresponding to the historical networkincidents identified by the artificial intelligence process may then beidentified, such as by retrieving the actions for the historical networkincidents from an actions database (e.g., one of the one or moredatabases 124) that includes information associated with actions takento resolve the alarms corresponding to the historical network incidents.The set of actions may represent a set of candidate actions that may beperformed to resolve the network incident that triggered generation ofthe alarm message.

A score may be determined for each of the one or more candidate actions,where the score represents a likelihood that a particular candidateaction will resolve the cause of the alarm message. The score may bedetermined based on a degree of similarity between the parameters of thealarm message and the historical network incidents identified by theartificial intelligence process. For example, the score may bedetermined based on a metric representing the degree of closenessbetween the parameters of a received alarm message and one or morenetwork incidents of a cluster determined to be similar to the incidentassociated with the alarm message. In an aspect, the score may representa confidence interval.

Once the candidate actions are determined and the scores are calculated,the fault mitigation device 110 may be configured to determine whetherto provide a notification regarding the alarm message to the user device140 or perform the candidate action(s) automatically. For example, wherethe score for a candidate action satisfies a threshold score, the faultmitigation device 110 may automatically execute at least one candidateaction to resolve the cause of the alarm message, but where the scorefor the candidate action does not satisfy the threshold score, the faultmitigation device 110 may transmit a notification to the user device 140that identifies the action. The threshold score may correspond to ascore that indicates a very high probability or likelihood theassociated candidate action will resolve a suspected cause of the alarmmessage (e.g., a 95% chance or higher, a 90% chance or higher, a 85%chance or higher, etc.), such that actions associated with scoresgreater than or equal to the threshold score may be automaticallyexecuted and actions associated scores less than the threshold score maybe transmitted to a user via the notification. In some aspects, allactions may be transmitted to the user device 140 for review by a userprior to the action being executed.

When the action is communicated to a user via the notification, thenotification may be presented to the user in a graphical user interface.The interface may enable the user to view the alarm message parameters,the parent or primary cause of the alarm message, the action suggestedto resolve the alarm, and the score. The interface may also provideinteractive elements to execute the action identified in thenotification. For example, if the action is to restart a device that iscausing the alarm, the notification may be presented within theinterface with interactive elements that may be selected or activated bythe user to restart the device. It is noted that the actions used toresolve network incidents may be executed remote from the source. Toillustrate, the user device 140 may located at a first location and thenode of the network causing the alarm may be located at a location thatis geographically remote from the first location. In such cases,activation of the interactive element presented at the graphical userinterface may cause one or more commands to be transmitted over anetwork to the node that caused the alarm message to be created. The oneor more commands may correspond to commands to execute the actionidentified by the artificial intelligence process, as described above.

In an aspect, the fault mitigation device 110 may be configured tocategorize the network incident associated with the alarm prior totransmitting a notification to the user device 140 or automaticallyexecuting any candidate actions. For example, the fault mitigationdevice 110 may be configured to classify the network incident as one of:a no trouble found (NTF) category, a self-healable category, and anon-self-healable category. The NTF category may be used for networkincidents arising from problems that occur due to minor fluctuations innetworks or interconnections which are (typically) found okay (e.g.,after analysis). As an example of an NTF fault, a flapping fault may bethe result of a loose interface connection which triggers multiplefaults or a spike in performance parameters such as CPU, memory, etc.that does not have an impact on a node or cause the node to becomedefunct. The self-healable known issues category may be used for networkincidents associated with recurring issues in the network for which aresolution is known and may be executed remotely (e.g., without havingto physically be present at the node of the network). For example, arogue process running in the network equipment or node is an example ofa self-healable fault and can be remotely stopped or killed by loggingin through the command line interface or via execution of an automatedscript configured to terminate the rogue process. The non-self-healablecategory may be used for recurring network issues for which theresolution may be known but cannot be executed without physicallytroubleshooting the cause of the issue(s) or being present at the node.Non-self-healable network incidents may require analysis andtroubleshooting by a user (e.g., an IT or network administrator) orreplacement/repair of hardware components of the node. It is noted thatthe exemplary categories described above have been provided for purposesof illustration, rather than by way of limitation and that othercategories and classifications may be utilized in accordance with theconcepts disclosed herein, such as a category associated with unknownnetwork incidents or a category for network incidents for which there isno known action that may be taken to resolve the network incident.

When categorization is utilized, the categorization of a networkincident may be used to determine how the action(s) is executed. Forexample, if the network incident is categorized as NTF, the faultmitigation device 110 may evaluate the network or interconnectionsassociated with the node or nodes associated with the alarm message andclear the alarm if the network incident was simply the result of minorfluctuations in the network (e.g., the node(s) or network connectionsappear to be operating correctly when evaluated by the fault mitigationdevice 110). If the fault mitigation device 110 is not able to analyzethe network or interconnections, the fault mitigation device 110 maytransmit a notification associated with the alarm message to the userdevice 140 and the user may analyze the contents of the notification andthe relevant portions of the network to determine whether to clear thealarm. Self-healable known issue network incidents may be handled in asimilar fashion. For network incidents categorized as non-self-healable,the fault mitigation device 110 may identify a specific user or team ofusers that should be notified of the network incident and may transmit anotification to the identified user(s). The identified users may bedetermined based on whether those users have experience resolving theidentified network incident. For example, the historical data may beanalyzed to determine one or more users that have handled similarprevious network incidents. Determining the user(s) based on analysis ofhistorical network incidents may enable the user(s) responsible forresolving the network incident to be identified more rapidly (e.g., ascompared to having a supervisor manually inquire as to which members ofthe IT or network administration team have appropriate experienceresolving the network incident, as is currently done in industrypractice) and may result in the network incident being resolved morequickly and efficiently.

It is noted that the categorization of network incidents may also beutilized in combination with the scoring concept described above. Forexample, a NTF or self-healable network incident may be automaticallyexecuted by the fault mitigation device 110 if the score associated witha candidate action satisfies the threshold score, but actions notsatisfying the threshold score may transmitted via one or morenotifications to users for confirmation (e.g., via the interactiveelements of the interface in which the notification is displayed) priorto executing any actions.

In an aspect, where more than one candidate action for resolving anetwork incident is identified, the notification may identify themultiple actions and the user (or fault mitigation device 110) mayperform the actions one at a time until the network incident isresolved. For example, the candidate actions may be ranked based ontheir respective scores (or another metric) and then executed (e.g., bya user via interactive elements presented in the interface orautomatically by the fault mitigation device 110) according to therankings. In such a scenario, the highest ranked candidate action may beexecuted first and an evaluation may be performed to determine whetherthat candidate action resolved the network incident. If the incident wasresolved, the alarm may be cleared and no further candidate actions maybe executed. If, however, the incident was not resolved, the nexthighest ranked candidate action may be executed and evaluated todetermine whether the incident was resolved by the second action. Thisprocess may continue until either all candidate actions have beenexecuted or the network incident is resolved. If all candidate actionsare executed and the network incident is not resolved, the networkincident may be referred to a user for manual investigation andresolution of the network incident (e.g., similar to thenon-self-healable process described above).

In aspects, as network incidents are detected and resolved by the system100, information associated with those resolved network incidents,whether resolved automatically or via technician intervention, may beincorporated into the historical data maintained by the one or moredatabases 124 and subsequently used to evaluate future networkincidents. For example, suppose that a network incident that has notbeen encountered previously is observed by the fault mitigation device110. The network incident may be evaluated using the above-describedtechniques, but since there are no known actions to resolve the networkincident, it may be referred to a user for manual resolution. Onceresolved, information associated with the network incident, such as theparameters of the alarm message generated in response to detection ofthe network incident and actions taken to resolve the network incident,may be recorded to the historical database. Subsequently, this new datamay be incorporated in the clustering and modelling processes describedabove, which may allow future instances of that network incident to beresolved automatically using the above-described techniques (e.g.,assuming the resolution is not related to a hardware failure thatrequires a technician to visit the node and replace a physicalcomponent). As more occurrences of that network incident occur, theactions taken to resolve it may also be refined, such as if a bettersolution is identified for resolving the network incident, therebyallowing the system 100 to more accurately identify actions that may betaken to resolve network incidents and allowing network incidents to beresolved more rapidly.

The learning capabilities of the fault mitigation device 110 and thediagnosis and resolution processes described above may result inimproved performance of the network(s) and the services and devicessupported by the network(s) (e.g., less network down time, increasedservice availability, etc.). Additionally, using information aboutresolved network incidents as feedback into the machine learning andartificial intelligence processes described above allows similaritiesbetween different network incidents to be identified. It may also allowactions used to resolve one network incident to be identified ascandidate actions for resolving network incidents sharing similarparameters but involving different nodes (or connections) within thenetwork(s).

Referring to FIG. 4, a block diagram illustrating aspects of faultmitigation processing in accordance with aspects of the presentdisclosure is shown. It is noted that the representation of the faultmitigation processing illustrated in FIG. 4 are intended to provide abetter understanding of the operations of the system 100 of FIG. 1 andthe various ways in which network incident data may be processed inaccordance with the concepts disclosed herein. In aspects, the exemplaryprocessing of FIG. 4 may be performed by the system 100 of FIG. 1, suchas by the fault mitigation device 110 of FIG. 1. The functionalitydescribed with respect to the various logical blocks shown in FIG. 4 maybe provided by one or more processors (e.g., the one or more processors112 of FIG. 1) or via software stored as instructions (e.g., theinstructions 122 of FIG. 1) that may be executed by a processor toperform the operations described below.

As shown in FIG. 4, fault mitigation processing in accordance with thepresent disclosure may be initiated upon receiving fault data 402. Thefault data 402 may be an alarm message, as described above withreference to FIG. 1 and may be received from a node of a network (e.g.,one of the nodes 132, 134, 136, 138 of the one or more networks 130 ofFIG. 1), from a device supported by the network (e.g., one of the otherdevices 150 of FIG. 1 or the user device 140 of FIG. 1, such as a user'ssmartphone device). Upon receiving the fault data 402, faultidentification processing may be performed, at block 410. The faultidentification processing may be configured to access informationassociated with known faults from a KEDB (e.g., one of the one or moredatabases 124 of FIG. 1) and determine whether the fault data 402identifies one of the known faults recorded to the KEDB. If the faultdata 402 identifies or corresponds to one of the known faults recordedto the KEDB (e.g., the outcome of fault identification processing is“Yes”), processing may proceed to block 470, where action evaluationprocessing is performed. The action evaluation processing may beconfigured to determine one or more actions to resolve the faultidentified in the fault data 402. Exemplary aspects of the actionevaluation processing 470 are described in more detail below.

If the fault data 402 does not identify or correspond to one of theknown faults recorded to the KEDB (e.g., the outcome of faultidentification processing 410 is “No”), input parameters 414 may beprovided to block 420 for clustering processing. The input parameters414 may include the parameters described above with reference to FIG. 1(e.g., node parameters, agent parameters, summary parameters, alert keyparameters, and alert group parameters). The clustering processing maybe configured to generate one or more clusters of network incidentsbased on historic network incident data associated with previous networkincidents, as described above with reference to clustering logic 114 ofFIG. 1. For example, the clustering processing may be configured togenerate groups of network incidents sharing similar combinations ofparameters based on historic network incident data and compare theparameters 414 to historic network incident data to identify networkincidents sharing similar parameters to the input parameters 414. Insome aspects, the clustering processing may be iteratively performed togenerate clusters based on the historic network incident data until itis divided into sufficiently distinct clusters, where each clusterrepresents a group of network incidents of a similar type or faultcause. Once the clusters are identified, the input parameters 414 may beanalyzed against the clusters to identify a cluster that is closest tothe input parameters 414. Identifying the cluster closest to the inputparameters 414 may result in identification of historic network faultsthat have been encountered previously and information associated withthe network faults of the identified cluster may be used to determineactions that may be taken to resolve the cause of the fault associatedwith the fault data 402.

Once the cluster is identified, fault resolution processing may beperformed, at block 430. The fault resolution processing may beconfigured to analyze the network incidents corresponding to the clusterto determine if a resolution to the network fault associated with thecluster exists, such as by analyzing the data associated with thenetwork incidents or faults of the cluster to determine whether actionstaken to resolve the network faults corresponding to the identifiedcluster are known. If a resolution to the network fault exists (e.g.,the outcome of fault resolution processing is “Yes”), the actionsutilized to resolve the previous network incidents may be identified andprovided to the action evaluation processing, at block 470. If aresolution to the network fault does not exist (e.g., the outcome offault resolution processing is “No”), processing may proceed to NTFanalysis, at block 440. The NTF analysis may be configured to determinewhether the network fault can be classified as an NTF network fault(i.e., no trouble found). If the network fault is classified as an NTFnetwork fault (e.g., the outcome of NTF analysis is “Yes”), processingmay proceed to the action evaluation processing, at block 470.

If the network fault is not classified as an NTF network fault (e.g.,the outcome of NTF analysis is “No”), processing may proceed toself-healable analysis, at block 450. The self-healable analysis may beconfigured to determine whether the network fault associated with thefault data 402 is self-healable. Self-healable network faults may benetwork faults that can be fixed remotely or without a technicianvisiting the node associated with the fault, such as by sending commandsto the node via a network. If the network fault is classified asself-healable (e.g., the outcome of the self-healing analysis is “Yes”),processing may proceed to the action evaluation processing, at block470.

If a resolution to the network fault is unknown (e.g., the outcome offault resolution processing is “No”) and the network fault associatedwith the fault data 402 is not classified as a NTF network fault or aself-healable network fault (e.g., the outcomes of NTF analysis andself-healable analysis are “No”), processing may proceed to block 460for diagnostic processing. The diagnostic processing may be configuredto perform diagnostic testing of the node to determine if the root causeof the network fault associated with the fault data 402 can beidentified. For example, the diagnostic testing may involve obtaininginformation associated with a current configuration of the node (e.g.,current software version, hardware components, etc.), performanceinformation associated with the node (e.g., current and/or historictraffic experienced by the node, etc.), change logs associated withchanges made to the node (e.g., software update history, replacement ofcomponents of the node, etc.), system logs, network devices, analysis ortesting of topological interconnection among the network devices,analyzing the types of services being supported by network nodes ordevices, and the like. The information obtained via the diagnostictesting may be analyzed to determine if a root cause of the networkfault can be identified. If the root cause can be identified, one ormore actions may be determined to resolve the cause of the networkfault.

The action evaluation processing, at block 470, may be configured todetermine one or more candidate actions for resolving the network fault.For example, where the fault is determined to be an NTF network fault,the action evaluation processing may determine that no action isnecessary and may clear the alarm associated with the fault data 402.This is because NTF network faults may not actually be faults and mayinstead just be temporary disruptions of a network (e.g., due to asudden and temporary spike traffic volume) that may be resolvedautomatically (e.g., as the sudden spike in traffic volume experiencedby a network node subsides). If the fault is determined to beself-healable, the one or more candidate actions may be determined basedon actions taken to resolve previous occurrences of similar faults andscores may be assigned to each candidate action. Because self-healablefaults may be resolved remotely, the action evaluation processing mayselect one of the one or more candidate actions for execution andprovide the selected candidate action to block 480 for actionprocessing. The action processing may be configured to transmit commandsto one or more nodes of the network and the command(s) may be configuredto cause automatic execution of the selected action(s) determined by theaction evaluation processing, such as to remotely initiate a reboot ofthe node, restart a process that has failed at the node, update softwareof the node, rollback a state of the node (e.g., restoring software of anode to a previous software version following a fault caused by asoftware change), terminate a rogue process running on the node, orother types of operations.

If the action evaluation processing determines the fault associated withthe fault data 402 cannot be resolved automatically (i.e., is notself-healable or NTF), the one or more candidate actions may be providedto ticket processing, at block 490. The ticket processing may beconfigured to generate a notification or ticket that may be provided toa user (e.g., IT personnel) and may request that the user execute theaction(s) determined by the action evaluation processing to resolve thecause of the fault associated with the fault data 402. To illustrate,the ticket processing may be configured to receive one or more commandsdetermined by the action evaluation processing and generate a ticketthat may be placed in a queue or transmitted directly to the user. Theuser responsible for handling the ticket may then review the actions andthe fault data and perform the suggested actions to resolve the fault,such as to replace a failed hardware component or other action. It isnoted that a ticket may also be generated where the diagnosticprocessing, at block 460, is unable to determine actions to resolve thenetwork fault and in such instances the user responsible for the ticketmay need to do further investigation to determine the cause of thenetwork fault and possible actions to resolve it.

Where multiple actions are identified for resolving a network fault, theaction evaluation processing may utilize the scores associated with eachcandidate action to determine an order in which the candidate actionsshould be executed. After a highest ranked or scored candidate action isexecuted the action evaluation processing may determine whether theexecuted action resolved the root cause of the network fault and ifresolved, may clear the network fault associated with the fault data402. If the executed action did not resolve the root cause of thenetwork fault, a next highest ranked or scored action may be executedand this process may continue until all candidate actions have beenperformed or the network fault is resolved. In some instances, none ofthe candidate actions may result in the cause of the network fault beingresolved and in such instances, a ticket may be issued (e.g., via ticketprocessing at block 490) so that a user may further investigate thefault and determine an action to resolve it.

As shown above, the process illustrated in FIG. 4 enables a system toleverage machine learning and artificial intelligence techniques torapidly identify causes of network incidents and determine actions thatmay be executed (e.g., locally or remotely) to mitigate the causes ofthe network incidents. The artificial intelligence techniques may beused to analyze historic network incidents data to identify networkincidents or faults that are similar to a newly occurring network faultand then determine action(s) that may be executed to resolve cause ofthe network incident. Additionally, the processing illustrated in FIG. 4enables diagnostics to be performed dynamically on network nodes todetermine causes of faults as well as monitoring of nodes of a networkto verify that actions taken to resolve network faults actually solvethe issues that caused the network fault (and execute additional actionswhen the issues are not solved). These features improve network faultmitigation systems by enabling network faults to be resolved morequickly, reducing network downtime, improving network serviceavailability, and providing such systems with the ability to diagnoseand mitigate new faults that may be incorporated into the artificialintelligence processing so that future occurrences of those faults maybe automatically corrected in the future.

Referring to FIG. 5, a flow diagram of a method for diagnosing andresolving network incidents according to embodiments of the presentinvention is shown as a method 500. In aspects, the method 400 may beperformed by a device, such as the fault mitigation device 110 ofFIG. 1. Steps of the method 400 may be stored as instructions (e.g., theinstructions 122 of FIG. 1) that, when executed by one or moreprocessors (e.g., the one or more processors 112 of FIG. 1), cause theone or more processors to perform the steps of the method 500. It isnoted that the method 500 may also incorporate the concepts describedabove with reference to FIGS. 2-4.

At step 510, the method 500 includes generating, by one or moreprocessors, a model of historic network incidents. As described abovewith reference to FIG. 1, the model may be generated using modellinglogic and may be used to evaluate network incidents that occur in anetwork, such as to identify historic network incidents that sharesimilarities with a current network incident, such as historic networkincidents that were caused by the same factors as the current networkincident. In aspects, clustering logic may be used, at least in part, togenerate the model, as described above. At step 520, the method 500includes receiving, by the one or more processors, an alarm messagecomprising information indicative of a network incident that occurred ina network (e.g., a real-time occurrence of a network incident). Asdescribed above with reference to FIG. 1, the information indicative ofthe network incident may include a plurality of parameters.

At step 530, the method 500 includes executing, by the one or moreprocessors, machine learning logic against the information indicative ofthe network incident and the model to determine one or more candidateactions. In an aspect, the method 500 may include executing clusteringlogic against historic network incident data, as described above withreference to FIG. 1. The clustering logic may be configured to identifya plurality of clusters associated with the network incidentsrepresented by the historic network incidents data, where each clusterof the plurality of clusters may corresponding to a set of historicnetwork incidents associated with a same network incident cause (e.g., anetwork configuration change, as described with reference to cluster 310of FIG. 3). In an aspect, the machine learning logic may be configuredto identify a cluster corresponding to the network incident based onsimilarities between the network incident and a set of historic networkincidents corresponding to the cluster. For example, example, theclustering logic may include (i.e., implement) a supervised machinelearning algorithm, such as the nearest neighbors algorithm describedwith reference to FIG. 1. The similarities may be determined based onidentification of a cluster of historic network incidents sharingsimilar values for the plurality of parameters included in the alarmmessage. As described above with reference to FIG. 1, the one or morecandidate actions determined in step 530 may be predicted to resolve acause of the network incident. For example, historic network incidentsin the identified cluster may be analyzed to determine actions taken toresolve the historic network incidents, which each share a similar causeto the network incident associated with the alarm message.

At step 540, the method 500 includes executing, by the one or moreprocessors, at least one candidate action of the one or more candidateactions. As described above with reference to FIG. 1, the at least onecandidate action may be executed automatically (e.g., by the faultmitigation device 110 of FIG. 1), or a notification (e.g., a messagethat includes information that identifies the one or more candidateactions) may be generated and transmitted to a user device and the atleast one candidate action is executed in response to an input receivedfrom the user device. When executed based on the input, the input maycorrespond to activation of interactive elements presented within agraphical user interface of the user device, as described above withreference to FIG. 1. In an aspect, the at least one executed action maybe determined based on a score assigned to the one or more candidateactions, based on a classification of the assigning a score for eachcandidate action of the one or more candidate actions, or based on bothscores and the classification(s), as described above.

In an aspect, the method 500 may also include monitoring a networkassociated with the network incident to determine whether the at leastone action that was executed (e.g., in step 440) resolved the cause ofthe network incident. If it is determined, based on the monitoring, thatthe at least one action did not resolve the cause of the networkincident, another action of the one or more candidate actions may beexecuted. This process may continue until all candidate actions havebeen executed or the network incident has been resolved, whichever comesfirst.

In an additional aspect, the method 500 may include generating feedbackdata based on the at least one action executed to resolve the cause ofthe network incident. As described above, the feedback data may be usedto update the historic network incident data and the model may betrained based on the feedback data. Incorporating the feedback data intothe model may enable candidate actions to be identified more accurately(e.g., candidate actions may be more likely to resolve occurrences ofnetwork incidents) and new network incidents may be more readilyintegrated into the processes of the method 500 to mitigate futureoccurrences of those network incidents.

As shown above, the method 500 provides a technique for rapididentification of causes of network incidents and for automaticallydetermining actions that may be executed (e.g., locally or remotely) tomitigate the causes of the network incidents. Using machine learning andartificial intelligence techniques, the method 500 enables historicnetwork incidents data to be leveraged to determine a cause of a networkfault and the action(s) that may be executed to resolve the cause of thefault. Additionally, the method 500 provides mechanisms for monitoringthe network for a period of time after execution of the action to ensurethe cause of the network incident is resolved and execute additionalactions if previous actions were unsuccessful in mitigating the cause ofthe network incident. The method 500 enables network faults to beresolved more rapidly and allow emerging network incidents (e.g.,network incidents that have not previously occurred or occurinfrequently) to be identified, resolved, and taken into considerationin the future, which may play a critical role in resolving issues thatoccur as network technologies continue to evolve and become morecomplex.

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made herein without departing from the spirit andscope of the invention as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods and steps described in the specification. Asone of ordinary skill in the art will readily appreciate from thedisclosure of the present invention, processes, machines, manufacture,compositions of matter, means, methods, or steps, presently existing orlater to be developed that perform substantially the same function orachieve substantially the same result as the corresponding embodimentsdescribed herein may be utilized according to the present invention.Accordingly, the appended claims are intended to include within theirscope such processes, machines, manufacture, compositions of matter,means, methods, or steps.

Moreover, the scope of the present application is not intended to belimited to the particular embodiments of the process, machine,manufacture, composition of matter, means, methods and steps describedin the specification.

1. A method for diagnosing and resolving network incidents, the methodcomprising: generating, by one or more processors, a model of historicnetwork incidents; receiving, by the one or more processors, an alarmmessage comprising information indicative of a network incident thatoccurred in a network; executing, by the one or more processors, machinelearning logic against the information indicative of the networkincident and the model to determine one or more candidate actions, theone or more candidate actions determined to resolve a cause of thenetwork incident; automatically assigning, by the one or moreprocessors, a score to each candidate action of the one or morecandidate actions, wherein each score represents a confidence intervalthat a corresponding candidate action will resolve the cause of thenetwork incident automatically executing, by the one or more processors,at least one candidate action of the one or more candidate actions basedat least in part on scores assigned to the at least one candidate actionand a threshold and prior to generation of a notification thatidentifies the one or more candidate actions, wherein the at least onecandidate action is executed to resolve a cause of the network incident;and executing, by the one or more processors, a second candidate actionof the one or more candidate actions upon a determination that executionof the at least one candidate action failed to resolve the cause of thenetwork incident, the determination based on monitoring a networkassociated with the network incident after execution of the at least onecandidate act.
 2. The method of claim 1, further comprising: executingclustering logic against historic network incident data, wherein theclustering logic is configured to identify a plurality of clustersassociated with the network incidents represented by the historicnetwork incidents data, each cluster of the plurality of clusterscorresponding to a set of historic network incidents associated with asame or similar network incident cause. wherein the model is generatedbased on the plurality of clusters, and wherein the machine learninglogic is configured to determine the one or more candidate actions toresolve the network incident via analysis of a portion of the modelusing parameters derived from the network incident.
 3. The method ofclaim 1, further comprising executing clustering logic against historicnetwork incident to identify a plurality of clusters, each cluster ofthe plurality of clusters corresponding to a set of network incidentsassociated with a same or similar cause, and wherein executing, by theone or more processors, the machine learning logic against theinformation indicative of the network incident and the model todetermine one or more candidate actions comprises: identifying, based onthe model, network incidents corresponding to a cluster sharingsimilarities with the network incident, and wherein the one or morecandidate actions are determined based on analysis of the historicnetwork incident data associated with the network incidents of theidentified cluster 4.-5. (canceled)
 6. The method of claim 1, furthercomprising: transmitting, to a user device, the notification thatincludes information that identifies the one or more candidate actions;and executing a second candidate action of the one or more candidateactions in response to an input received from the user device.
 7. Themethod of claim 6, wherein the input corresponds to activation of aninteractive element presented within a graphical user interface of theuser device.
 8. (canceled)
 9. The method of claim 1, further comprising:generating feedback data based on the at least one candidate actionexecuted to resolve the cause of the network incident; and updatinghistoric network incident data based on the feedback data.
 10. Themethod of claim 9, further comprising training the model based on theupdated historic network incident data.
 11. (canceled)
 12. The method ofclaim 1, further comprising determining a classification for the networkincident, wherein the classification is selected from a plurality ofclassifications, and wherein executing the at least one candidate actionis based at least in part on the classification of the network incident.13. A system comprising: a memory; and one or more processorscommunicatively coupled to the memory and configured to: generate amodel of historic network incidents; receive an alarm message comprisinginformation indicative of a network incident that occurred in a network;execute machine learning logic against the information indicative of thenetwork incident and the model to determine one or more candidateactions, the one or more candidate actions determined to resolve a causeof the network incident; automatically assign a score to each candidateaction of the one or more candidate actions wherein each scorerepresents a confidence interval that a corresponding candidate actionwill resolve the cause of the network incident automatically execute atleast one candidate action of the one or more candidate actions based atleast in part on scores assigned to the at least one candidate actionand a threshold and prior to generation of a notification thatidentifies the one or more candidate actions, wherein the at least onecandidate action is executed to resolve a cause of the network incident;and execute a second candidate action of the one or more candidateactions upon a determination that execution of the at least onecandidate action failed to resolve the cause of the network incident,the determination based on monitoring a network associated with thenetwork incident after execution of the at least one candidate action.14. The system of claim 13, wherein generating the model of the historicnetwork incidents comprises: executing clustering logic against historicnetwork incident data to identify a plurality of clusters associatedwith network incidents represented by the historic network incidentdata, each cluster of the plurality of clusters corresponding to a setof network incidents associated with a same or similar root cause, andwherein execution of the machine learning logic against the informationindicative of the network incident and the model to determine one ormore candidate actions comprises: identifying, based on the model,network incidents corresponding to a cluster sharing similarities withthe network incident, and wherein the one or more candidate actions aredetermined based on analysis of the historic network incident dataassociated with the network incidents of the identified cluster.
 15. Thesystem of claim 14, wherein the model is generated by iterativelyexecuting the clustering logic against the historic network incidentdata.
 16. (canceled)
 17. The system of claim 13, wherein the one or moreprocessors are configured to: generate feedback data based on the atleast one candidate action executed to resolve the cause of the networkincident; update historic network incident data based on the feedbackdata; and update the model based on the updated historic networkincident data.
 18. A non-transitory computer-readable storage mediumstoring instructions that, when executed by one or more processors,cause the one or more processors to perform operations comprising:generating a model of historic network incidents; receiving an alarmmessage comprising information indicative of a network incident thatoccurred in a network; executing machine learning logic against theinformation indicative of the network incident and the model todetermine one or more candidate actions, the one or more candidateactions determined to resolve a cause of the network incident;automatically assigning a score to each candidate action of the one ormore candidate actions wherein each score represents a confidenceinterval that a corresponding candidate action will resolve the cause ofthe network incident automatically executing at least one candidateaction of the one or more candidate actions based at least in part onscores assigned to the at least one candidate action and a threshold andprior to generation of a notification that identifies the one or morecandidate actions, wherein the at least one candidate action is executedto resolve a cause of the network incident; and executing a secondcandidate action of the one or more candidate actions upon adetermination that execution of the at least one candidate action failedto resolve the cause of the network incident, the determination based onmonitoring a network associated with the network incident afterexecution of the at least one candidate action.
 19. The non-transitorycomputer-readable storage medium of claim 18, the operations furthercomprising: executing clustering logic against historic network incidentto identify a plurality of clusters, each cluster of the plurality ofclusters corresponding to a set of network incidents associated with asame or similar root cause, and wherein executing, by the one or moreprocessors, the machine learning logic against the informationindicative of the network incident and the model to determine one ormore candidate actions comprises: identifying, based on the model,network incidents corresponding to a cluster sharing similarities withthe network incident, and wherein the one or more candidate actions aredetermined based on analysis of the historic network incident dataassociated with the network incidents of the identified cluster.
 20. Thenon-transitory computer-readable storage medium of claim 18, theoperations further comprising: after executing the at least onecandidate action, generating feedback data based on results of executingthe at least one candidate action to resolve the cause of the networkincident; updating historic network incident data based on the feedbackdata; and training the model based on the updated historic networkincident data.
 21. The method of claim 1, wherein the informationindicative of the network incident comprises an alert key parameter andan alert group parameter.
 22. The method of claim 1, wherein the secondcandidate action is a highest-ranked candidate action of the one or morecandidate actions remaining after execution of the at least onecandidate action, and wherein the scores are assigned to the one or morecandidate actions are ranked based further on categorization of thehistoric network incidents associated with the one or more candidateactions, and wherein a score assigned to a candidate action associatedwith a first category of historic network incidents is different than ascore assigned to a candidate action associated with a second categoryof historic network incidents.
 23. The method of claim 1, furthercomprising: determining, by the one or more processors, whetherexecution of all of the one or more candidate actions failed to resolvethe cause of the network incident and issuing, by the one or moreprocessors, a ticket associated with the network incident based on adetermination that execution of all of the one or more candidate actionsfailed to resolve the cause of the network incident.
 24. The method ofclaim 1, further comprising generating a notification that identifiesany of the one or more candidate actions that are associated with ascore that does not satisfy the threshold.